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ABSTRACT 


The  visual  looming  effect,  i.e.,  expansion  of  object’s  size  in  the  retina,  has  been 
shown  to  be  very  important  when  action  is  taking  place.  Most  existing  research  work 
on  "looming"  has  been  done  by  psychologists  and  is  qualitative  or  limited-quantitative. 
In  order  to  use  the  existing  results  for  robotics  applications,  a more  mathematically- 
oriented  approach  for  looming  is  needed.  In  this  paper  we  take  a quantitative  approach 
to  understanding  looming.  First  we  define  looming  mathematically  and  show  geometri- 
cal properties  of  objects  that  produce  the  same  value  of  looming.  Then  we  explain 
how  to  measure  looming  in  the  general  case  of  motion  using  optical  flow  and  the 
change  of  the  projection  of  a 3-D  object  on  the  retina.  It  is  shown  how  a multiresolu- 
tion logarithmic  retina  simplifies  the  measurement  of  looming,  and  how  the  results  can 
be  combined  with  previous  work  on  visual  fields.  We  suggest  a new  representation  of 
space  based  on  looming  and  the  so  called  Equal  Flow  Circles  (EFCs). 


This  research  was  supported  in  part  by  a grant  from  the  National  Science  Founda- 
tion to  Florida  Atlantic  University  (Grant  #IRI-9 115939). 


1.  INTRODUCTION 


The  Random  House  dictionary  defines  looming  as:  "Coming  into  view  in  indistinct 
and  enlarged  form;  rising  before  the  vision  with  an  appearance  of  great  or  portentous 
size".  Although  this  definition  is  a qualitative  one,  it  is  clear  that  it  deals  with  expan- 
sions of  objects  on  the  retina.  Usually,  expansion  of  object’s  size  on  the  retina  is  a 
result  of  a decreasing  distance  over  some  period  of  time  between  the  observer  and  the 
object. 

The  perception  of  looming  which  is  an  indication  for  possible  collision  is  critical 
to  survival  for  creatures  of  nature.  It  is  necessary  for  locomotion  in  a complex  natural 
3-D  world.  The  reaction  to  this  visual  stimulus  is  the  result  of  some  kind  of  "perceived 
threat",  i.e.,  the  measured  relative  rate  of  expansion  of  objects  on  the  retina  corresponds 
to  a visual  timing  parameter  that  causes  the  subject  to  defensively  react  to  reduce  the 
visual  threat. 

Looming  and  looming-related  actions  have  been  studied  by  many  researchers, 
mainly  psychologists  [1-24],  but  most  of  the  work  is  qualitative  or  limited-quantitative 
[9].  The  looming  effect  [20,45],  which  is  the  result  of  optical  expansion  of  objects,  has 
shown  to  cause  defensive  reaction  in  several  animals  as  well  as  babies  [6,20].  In 
several  reported  experiments  the  subjects’  use  of  visual  information  in  the  last  few  hun- 
dreds of  milliseconds  before  a collision  occurs  was  examined. 

In  order  to  use  the  existing  results  for  robotics  applications,  a more 
mathematically-oriented  approach  for  looming  is  needed. 

Lee  [9,10]  and  his  colleagues  [11-14]  showed  how  to  quantitatively  measure  the 
"time-to-contact"  from  optical  flow  for  an  observer  who  undergoes  only  translation 
when  the  optical  axis  coincides  with  the  motion  vector.  Extensions  of  the  time-to- 
contact  concept  were  recently  reported  in  [25,26].  Our  definition  of  looming  is  related 
to,  but  different  from,  the  time-to-contact.  The  time-to-contact  according  to  [9]  is  the 
time  it  takes  for  an  observer  to  hit  a specific  plane  perpendicular  to  the  direction  of 
motion,  i.e.,  it  deals  with  the  relative  change  of  the  depth  (as  opposed  to  range). 
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Assume  an  observer  moving  in  a 3-D  environment  filled  with  several  balls  (Figure 
1).  During  the  motion  of  the  observer  relative  to  the  balls,  both  the  size  of  each  ball 
and  the  location  its  centroid,  when  projected  into  the  image,  are  continuously  changing. 
If  the  distance  between  a ball  and  the  center  of  the  camera  decreases  then  the  projected 
image  of  the  ball  will  increase  in  area.  The  relative  rate  of  expansion  of  the  area  over 
time  of  the  imaged  ball  causes  the  "looming"  effect  and  it  is  proportional  to  the  time 
derivative  of  the  relative  distance  (range)  between  the  observer  and  the  ball  divided  by 
the  relative  distance  (range)  itself.  The  looming  is  a measurable  variable  and  can  be 
extracted  directly  from  a sequence  of  2-D  images  using  optical  flow,  relative  change  in 
area,  etc. 

In  this  paper  we  approach  looming  from  quantitative  as  well  as  qualitative  points 
of  view.  We  start  with  a mathematical  definition  of  looming,  followed  by  an  expres- 
sion for  looming  in  vector  form.  We  show  that  looming  is  rotation  independent  and 
that  there  are  equal  looming  spheres  each  of  which  corresponds  to  a value  of  looming. 
In  other  words,  each  point  on  a particular  equal-looming  sphere  appears  as  having  the 
same  looming  regardless  of  the  instantaneous  rotation  of  the  camera.  We  introduce 
several  methods  for  measuring  looming,  using  optical  flow  as  well  as  the  expansion  of 
the  projection  of  objects  on  the  retina.  We  describe  camera-centered  coordinate  systems 
that  are  used  here,  and  derive  expressions  that  relate  optical  flow  and  distance  of  a 
point  to  six-degree-of-ffeedom  camera  motion.  When  the  optical  flow  constraint  (as 
described  by  Horn  and  Schunck  [36])  is  added  to  these  equations,  a pixel-based 
closed-form  solution  for  the  looming  of  a point  in  the  camera  coordinate  system  is 
obtained.  The  rate  of  expansion  of  projected  objects  is  another  method  introduced  to 
measure  looming. 

It  is  shown  that  by  using  a logarithmic  retina  to  measure  looming,  many  computa- 
tions become  significantly  simpler,  and  that  the  computations  are  independent  of  the 
object’s  size.  The  analysis  becomes  even  simpler  when  dealing  with  looming  and 
fixation.  A mapping  of  space  is  suggested  using  equal  looming  spheres  and  Equal 
Row  Circles  (EFCs,  as  described  in  [63]). 
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The  work  described  in  this  paper  is  highly  related  to  the  area  called  "Active 
Vision",  in  particular  purposive  quantitative  as  well  as  qualitative  vision,  since  it  can  be 
used  to  actively  control  the  viewer’s  parameters.  [30,31,33-35,57,66]. 

2.  MATHEMATICAL  DEFINITION  OF  LOOMING 


Let  a camera  and  an  object  (e.g.,  a ball)  move  arbitrarily  in  a 3-D  environment 
(Figure  2a).  Then  at  two  different  time  instants  they  may  be  in  different  relative  loca- 
tions. The  distance  (range)  from  the  center  of  the  ball  to  the  pinhole  point  of  the  cam- 
era may  change,  thus  resulting  in  different  sizes  of  projected  images.  Figure  2b  shows  a 
3-D  ball  and  its  images  at  two  different  time  instants.  The  looming  effect  is  caused  by 
the  expansion  of  the  projected  ball.  Intuitively,  the  relative  rate  of  expansion,  i.e.,  the 

A 2 —A  i 

relative  change  of  the  projection  of  the  ball  over  the  period  of  time  t2-tx  in 

A\ 

which  the  change  takes  place,  is  highly  related  to  the  relative  change  of  the  range 
/?!-/?  2 


R 


during  the  same  period  of  time.  (In  fact,  for  objects  that  are  small  relative  to 


their  range  from  the  observer,  they  are  proportional,  i.e.,  differ  by  a scale  factor.)  Thus 
one  can  define  looming  in  terms  of  the  relative  change  in  range  instead  of  relative 
change  in  the  object’s  projection. 


In  order  to  mathematically  define  looming  and  suggest  ways  to  measure  it  we 
shall  consider  infinitesimally  small  3-D  balls.  Let  the  relative  distance  between  the 
observer  and  a point  P (the  center  of  the  infinitesimally  small  ball)  at  time  instant  be 
R j and  at  time  instant  1 2 be  R2  (Figure  3).  Then,  we  define  looming  L as  the  nega- 
tive value  of  the  time  derivative  of  the  relative  distance  between  the  observer  and  the 
point  P , divided  by  the  relative  distance  R , i.e.. 


(R  2~R  i) 


L = 


- lim 

A/-»0 


A t 
*1 


0) 


where  At  = t2  - ty  Or 


(2) 
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where  dot  denotes  derivative  with  respect  to  time.  The  reason  for  the  negative  sign  in 
Equations  (1)  and  (2)  is  to  associate  image  expansion  with  positive  looming. 

This  definition  allows  the  use  of  the  term  "looming  of  a point"  for  describing  the 
—R 

value  of  L (i.e.,  — — ) of  a point 
R 

3.  LOOMING  IN  VECTOR  FORM 
3.1  STATIONARY  ENVIRONMENT 

Assume  a moving  observer  in  a stationary  environment  Let  the  instantaneous 
vector  from  the  observer  to  a stationary  point  P be  R,  the  instantaneous  translational 
velocity  vector  of  the  camera  be  t and  the  instantaneous  angular  velocity  vector  of  the 
camera  be  © (Figure  4).  Then  the  expression  for  the  velocity  vector  V of  point  P in 
camera  coordinates  is  (see  for  example  [51]): 

V = - t - co  x R (3) 


The  general  expression  for  the  looming  L in  vector  form  is: 


L 


VR 

RR 


or 


L 


VR 

I R I2 


(4) 


The  last  expression  is  identical  to  the  expression  in  Equation  (2)  for  the  following  rea- 

R 

sons:  is  a unit  vector  along  the  line  that  connects  the  observer  with  the  point  in 

R R 

3-D.  V'-j^y  is  the  projection  of  the  velocity  vector  V along  the  direction,  i.e.,  it 


is  R . Hence 


VR  R 


\R  I2  R ' 

By  substituting  (3)  into  (4)  the  expression  for  the  looming  L becomes: 
(-  t - © x R)  R 


L = - 


RR 


(5) 


(co  x R)  R = 0 (since  the  vector  © x R is  always  perpendicular  to  the  vector  R)  and 
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thus  expression  (5)  becomes: 


L = 


t R 
R R 


(6) 


In  the  expression  for  looming  (Equation  (6))  neither  the  distance  nor  the  velocity 
between  the  observer  and  the  point  in  space  need  to  be  explicitly  known.  Only  the 
ratio  between  the  two  dot  products  needs  to  be  known.  The  expression  is  time  depen- 
dent only,  i.e.,  L is  measured  in  [rime-1]  units. 

Note  that  the  result  for  L in  Equation  (6)  is  a scalar  which  is  dependent  on  the 
camera  translational  component  of  the  velocity  vector,  but  independent  of  any  camera 
rotation. 

3.2  MOTION  OF  A CAMERA  AND  POINTS 

In  the  case  where  both  the  camera  and  the  point  are  moving,  we  can  always 
analyze  it  as  if  the  point  is  stationary  and  the  camera  undergoes  translation  and  rota- 
tion. The  translation  vector  of  the  camera  relative  to  the  "artificially"  stationary  point 
is  clearly  the  result  of  subtracting  the  point  translation  vector  from  the  camera  transla- 
tion vector.  For  looming  computation  the  rotation  is  not  relevant,  and  thus  only  the 
relative  translational  information  between  the  observer  and  the  point  is  relevant  for  any 
kind  of  relative  motion  between  the  observer  and  the  point.  The  same  result  is  true  for 
motion  of  a camera  and  a 3-D  rigid  object. 

4.  EQUAL  LOOMING  SURFACES 

In  this  section  we  answer  the  following  question:  "What  is  the  location  of  points 
in  3-D  space  that  result  in  the  same  looming  value  L for  any  motion  of  the  camera?" 

4.1  GENERAL  MOTION  OF  CAMERA 

Let  us  set  the  value  of  the  looming  expression  (Equation  (6))  to  a constant  and 
obtain  the  equal  looming  surfaces.  If  the  angle  between  t and  R is  a then  Equation  (6) 
becomes: 

r It  I (H\ 

L = — cosa  (7) 

A 

where  It  I is  the  absolute  value  of  the  vector  t.  Without  loss  of  generality  attach  an 
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XYZ  coordinate  system  to  the  observer  such  that  the  Y axis  coincides  with  the  instan- 
taneous vector  t as  in  Figure  5.  In  this  case  cosa  = , — and  R - Vx2+y2+z2. 


Vjc2+y2+z2 


Substituting  these  expressions  into  Equation  (7)  yields: 


L = 


Itly 


x2+y2+z2 


(8) 


or 


x2  + (y 


It!  2 2 Jtl  2 

2L>  +Z  =(^ 


(9) 


Equation  (9)  is  an  equation  of  a sphere  whose  center  is  located  at  (0,— ,0)  and  its 

radius  is  ™~-.  The  center  of  the  sphere  lies  on  the  instantaneous  translation  vector, 

and  the  observer  lies  on  a point  of  this  sphere  (the  origin  of  the  coordinate  system  in 
Figure  5).  Note  that  this  sphere  corresponds  to  a particular  looming  value,  i.e.,  all 
points  on  this  sphere  result  in  the  same  value  of  L.  Depending  on  It!  this  sphere  will 
expand  or  shrink  in  3-D.  Figure  6 shows  points  in  3-D  that  produce  the  same  looming 
value,  and  Figure  7 shows  the  location  of  the  equal  looming  points  for  different  loom- 
ing values.  Each  circle  in  Figures  6 and  7 is  a section  of  an  equal  looming  sphere. 

4.2  POSITIVE-,  NEGATIVE-,  AND  ZERO-LOOMING  SURFACES 

Of  the  equal  looming  surfaces  some  correspond  to  positive  values  of  looming,  oth- 
ers correspond  to  negative  values  of  looming  and  there  is  a plane  (i.e.,  a sphere  with 
infinite  diameter)  that  corresponds  to  zero  looming.  Figure  8 shows  the  three  types  of 
surfaces.  Points  on  the  plane  which  is  perpendicular  to  the  instantaneous  translational 
vector  and  passes  through  the  pinhole  point  of  the  camera  produce  zero  looming. 
Points  on  the  hemisphere  in  front  of  the  moving  camera  (i.e.,  in  front  of  the  zero- 
looming  plane)  produce  positive  values  of  looming,  and  points  on  the  hemisphere 
behind  the  moving  camera  (i.e.,  behind  the  zero-looming  plane)  produce  negative  loom- 
ing values.  Note  that  the  smaller  the  sphere  the  higher  the  absolute  value  of  the  loom- 
ing. The  different  points  shown  in  Figure  8 correspond  to  different  L which  depends 
on  their  location.  Note  that  at  0 = 45°  the  looming  value  is  maximum  and  at  0 = -45° 
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it  is  minimum.  At  these  two  angles  the  size  of  the  sphere  is  minimal.  The  looming  as  a 
function  of  the  angle  0 (as  defined  in  Figure  8)  is  graphically  described  in  Figure  9. 

5.  LOOMING  AND  SCALED  SPACE 

As  has  been  shown  previously,  the  radii  of  the  equal  looming  spheres  are  func- 
tions of  time  (measured  in  reverse  unit  of  time)  and  are  not  measured  in  length  units. 
This  means  that  different  spheres  in  3-D  may  result  in  the  same  looming  value  when 
dealing  with  different  instantaneous  translational  velocity.  For  some  direction  0,  a far 
point  with  large  value  of  It  I may  have  the  same  value  of  L as  a close  point  with  small 

value  of  Itl.  In  the  scaled  space,  i.e.,  in  the  time  domain  where  the  coordinates  are 
X Y Z 

and  -jjy  (time-based  coordinates)  a single  sphere  represents  a single  looming 
value  regardless  of  the  exact  translational  velocity. 

6.  MEASUREMENT  OF  LOOMING 

We  show  two  basic  ways  for  measuring  looming.  One  is  based  on  optical  flow 
where  the  derivation  is  for  any  six-degree-of-freedom  motion  of  a camera  in  a station- 
ary environment.  The  second  way  is  based  on  the  relative  rate  of  expansion  of  the 
objects’  projections  and  holds  for  any  kind  of  relative  motion  between  the  camera  and 
the  object. 

6.1  USING  OPTICAL  FLOW 

6.1.1  EQUATIONS  OF  MOTION  AND  OPTICAL  FLOW 

This  section  describes  the  equations  that  relate  a point  in  3-D  space  to  the  projec- 
tion of  that  point  in  the  image  for  general  six-degree-of-freedom  motion  of  the  camera. 
The  following  derivation  is  for  motion  in  a stationary  environment.  Some  of  the  equa- 
tions are  well  known  [51,52]. 

Assume  a moving  camera  in  a stationary  environment  with  the  coordinate  system 
fixed  with  respect  to  the  camera  as  shown  in  Figure  10.  Assume  a pinhole  camera 
model,  such  that  the  pinhole  of  the  camera  is  at  the  origin  of  the  coordinate  system. 
We  derive  the  optical  flow  components  in  a spherical  coordinate  system  ( R 0<J>).  In  this 
frame,  angular  velocities  (0  and  <|))  of  any  point  in  space,  say  P,  are  identical  to  the 
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optical  flow  values  0 and  <j)  of  the  projection  of  P (i.e.,  P’)  in  the  image  domain  (Figure 

11). 

The  relationship  between  the  image  flow  values  in  the  spherical  domain  is  given 
by  the  following  equations.  Let  the  instantaneous  cartesian  coordinates  of  the  point  P 
be  R = (X  ,Y  Z )T  (where  the  superscript  T denotes  transpose).  Let  the  instantaneous 
translational  velocity  of  the  camera  be  t = (U  ,V  ,W)T  and  the  instantaneous  angular 
velocity  be  CD  = (AJ3,C)r.  Then  the  velocity  vector  V of  the  point  P with  respect  to 
the  XYZ  coordinate  system  is  as  shown  in  Equation  (3),  i.e., 

V = -t-(oxR 


or: 


yx  = U-BZ+CY 

(10) 

Vy  = - V-CX+AZ 

(ID 

Vz  = - W-AY+BX 

(12) 

where  Vx , Vy,  and  Vz  are  the  components  of  the  velocity  vector  V along  the  X,  Y, 
and  Z directions  respectively.  Let  sQ  = sin0,  c0  = cos0,  = sin<j),  = cos(J), 

and  R — iRl.  To  convert  from  XYZ  to  R 0$  coordinates  we  use  the  relations: 


X =R  c^Cq 

(13) 

Y = R c^Sq 

(14) 

Z ~ R 

(15) 

In  order  to  find  the  optical  flow  of  a 3-D  point  in  R 0<j)  coordinates,  we  use  the  follow- 
ing relations  and  transformations  (see  [52]  and  Figure  10).  Let  VR , Ve,  and  be  the 
components  of  the  vector  V in  spherical  coordinates,  then 


Vr 

1 o 

O 

o 

Cq  Sq  0 

Vb 

= 

0 1 0 

-Sq  c 0 0 

o' 

u 

o 

-©- 

r . 

0 0 1 

(16) 


Also  from  [52] 


9- 


VR  =/?, 

Ve^RQc^ 

V^  = R( i>, 

where  dot  denotes  derivative  with  respect  to  time. 

Using  equations  (10)-(19)  the  following  expressions  are  obtained: 


(17) 

(18) 
(19) 


/? 

C^C0  C^50  5^, 

- U-  B R Sty  + C R Cty  Sq 

/?  0 C^| 

= 

“•*0  C@  0 

-V-C  R CtyCQ  + A R Sty 

<j> 

~ Sty  Cq  —StySQ  C0 

-W-  A R Cty  Sq  + B R CtyCQ 

(20) 


There  are  four  unknowns  in  Equation  set  (20):  RJR,Q,  and  <{>.  For  each  pixel,  0 and  <J> 
are  known.  The  motion  parameters  (A  J2  ,C  ,U  ,V  ,W)  are  also  known. 

6.1 2 THE  OPTICAL  FLOW  CONSTRAINT 

If  the  brightness  / varies  smoothly  with  0,  <{),  and  t (time),  then  we  obtain  the  opt- 
ical flow  constraint  equation  using  the  chain  rule  (see  [36]): 


or 


where 


M.Q  + JL,j,  + iL  = o 

30  3<>  v 3 1 


^0  ® + ^6  4 > + 7/  = 0 


(21) 


(22) 


/«  = 


dl_ 

00 


/ =*- 
6 0<j> 


I = * 
''  Or' 


When  dealing  with  digital  images  /0,  /*,  and  It  can  be  approximated  from  the  image 
sequence  as  follows.  For  a given  pixel  (0,  ,<j)y- ) in  the  image  at  time  instant  tk 

0/  _ / (0;,<t>7 ,tk)-I 


00,  0,-0,  _! 

a/  _ i(Qi$j,tk)~  i(Qi$j-i,tk) 


0<D, 


+J-+J- 


(23) 

(24) 


0/  _ — / (Q,  ,4>y >^_i) 

dtk 


h~h- i 


(25) 
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6.1J  LOOMING  COMPUTATIONS 

Equation  set  (20)  together  with  equation  (22)  form  a set  of  four  equations  with 
four  unknowns  R,  R,  6,  and  «j>.  Solving  these  equations  for  R yields: 

(Us  q-Vc  q)Iq+c^(Us  tyC  e+Vs  ^ e-Wc  ^ 4, 

R — — — — — — — — — — — — — (26) 

“ csjA  ~ c ty(Bc  q—As  q)I (j,  + (Cc  fy—s  §(Bs  q+Ac  q))I @ 

Solving  for  R yields: 

R = -Uc^Cq-Vc^q-Ws^  (27) 

And  by  combining  Equations  (26)  and  (27)  we  get: 

l - _ A - _ (c'Uc  e~Vc  ^ Q~Ws  4>)(~  c^l~  c^BcQ-AsQVt>  + (Cc^-s^jBse+Ace))IB) 
R (Us  q-Vc  q)I  e+c  q(Us  ^ cQ+Vs  ^ e-Wc  ^)I ,,, 

(28) 

If  the  translation  vector  t forms  0=0,  and  <J)=<|),  angles  in  the  /?-©-4>  coordinates  (Fig- 
ure 10)  then  U ,V , and  W can  be  expressed  as: 


U = Itlc^CQ, 

(29) 

V = Itlc^se, 

(30) 

W = ltlj0< 

(31) 

Using  the  last  three  expressions.  Equation  (28)  can  be  rewritten  as: 

_ (c0lc0|c0ce+c0,s0<c$5e+s0,J0)  (-c^-c^Bcq-Asq)^  + (Cc^-s^Bsq+Acq))^) 

(c  C Q, s Q~C  $,s  Q,c  q)I  Q+c  $(C  $,C  Q,S  tyC  q+C  fyS  Q'S  ^ q~S  fyC  ^ 

(32) 

The  meaning  of  solution  (32)  is  that  if  the  location  of  the  pixel  in  the  image,  the 
instantaneous  rotational  motion  parameters  of  the  camera,  the  instantaneous  transla- 
tional direction  of  motion,  and  the  spatial  and  temporal  intensity-changes  (i.e.,  changes 
in  brightness)  are  given,  then  the  looming  of  the  corresponding  point  in  3-D  can  be 
obtained. 
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At  first  inspection,  33  multiplications  and  divisions,  and  11  summations  and  sub- 
tractions appear  to  be  necessary  for  each  pixel  at  each  instant  of  time.  However,  for  a 
given  pixel,  0 and  <|>  are  constants  and  therefore  all  functions  of  0 and  0 such  as  cQc^ 
can  be  precomputed.  Also,  the  rate  of  change  of  the  motion  parameters  A J3  ,C  and 
0,,0r  is  usually  much  lower  than  the  rate  of  change  of  the  image  variables  7e,  I § and 
It . Thus  the  update  rate  of  the  motion  parameters  can  be  significantly  smaller  than  the 
computation  of  L . For  example,  for  a translation  in  the  XY  plane,  and  rotation  about 
the  Z axis  (i.e..  A,  B,  and  W are  zero),  U,  V,  and  C may  be  constants  for  some 
period  of  time  but  7e,  7^  and  It  change.  Practically,  accelerations  (either  translational  or 
rotational)  are  limited  due  to  mass  (or  inertia)  constraints.  Thus,  any  function  of 
0,0,<J>,  ,0,  yA  yB  ,C  will  typically  have  an  update  rate  that  is  an  order  of  magnitude 
smaller  than  the  rate  of  change  of  the  pixel  intensity  (brightness). 


Equation  (32)  can  be  rewritten  as  follows: 
r 0O(a3  7,  + a4  / <t>  + a5  70) 

— T , I wi) 

ai  Iq  + ei2  I $ 

where 

a0=  c$lcQ,c$c&¥chsQ,c$sQ+shsQ  (34) 

d \ — C^Cq  Sq  C^Jq^Cq  (35) 

^ 2 t7(j)(C(j^ C Q'S (JjCq+C ^ q S §tC (36) 

a3  = - (37) 

a 4 = Cq(Asq-Bcq)  (38) 

a5  = Cc tf-s ^(Bs q+Ac e)  . (39) 


By  observing  equation  (33)  it  is  clear  that  for  given  7e,  7 ^ and  It , at  the  fast  sam- 
pling rate  there  are  only  six  multiplications,  one  division,  and  three  summations  per 
pixel.  (Note  the  the  number  of  multiplication  can  be  reduced  to  five  by  multiplying  a0 
by  a3,a4  and  a5 .) 
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For  each  point  in  space  there  is  a set  of  four  equations  that  result  in  a looming 
value  of  a point  in  space.  The  expression  for  L that  corresponds  to  a pixel  in  the  image 
can  be  computed  using  a pixel-based  processor,  and  special  purpose  computers  may  run 
in  parallel  to  compute  looming  values  of  all  visible  points  in  space.  For  each  pixel 
there  may  be  a "fast"  computer,  and  a "slow"  computer.  The  "fast  computer"  extracts 
/ q,  7 a,  and  It , and  computes  the  looming  L that  corresponds  to  this  pixel.  The  "slow" 
computer  generates  a0^1,a2»^3»^4  and  a5  from  Equations  (34)-(39)  at  a low  update 
rate.  A suggestion  for  a structure  of  such  a "multi-rate"  computer  can  be  found  in  [64]. 

The  problem  with  this  approach  is  that  the  optical  flow  is  highly  dependent  on 
both  the  translation  and  rotation  of  the  camera,  and  so  the  expression  for  the  looming  L 
in  Equation  (32)  contains  rotation  parameters. 

For  the  purpose  of  analyzing  looming  this  approach  is  quite  limited.  Practically 
there  are  other  severe  limitations  which  are  related  to  the  measurements  of  spatial  and 
temporal  gradients. 

6.1.4  SPECIAL  CASE  1 

In  order  to  intuitively  understand  expression  (32),  we  show  a special  case.  Sup- 
pose that  the  camera  undergoes  translation  U and  V in  the  instantaneous  XY  plane  and 
rotation  C about  the  Z axis  only,  i.e.,  A = B = W = 0.  Assume  <J>  = 0,  i.e.,  the  pixel 
which  is  analyzed  lies  in  the  XY  plane.  Substituting  these  values  in  Equation  (32) 
results  in  the  following  expression  for  L : 

(c  0 c 0+S  0 s o)(-It  + Cl q) 

L = (40) 

(CqSq-SqCq)I  0 


or  (assuming  7e* 0) 

(ce(ce+-se,iSe)("_®  + C) 
L = 

(c  Q,s  Q~s  Q,c  q) 


(41) 


where 


Given  that  0*C , Equation  (41)  is  a solution  for  any  point  in  the  XY 


plane  when  translating  in  this  plane  and  rotating  about  the  Z axis. 
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6.1.5  SPECIAL  CASE  2 


For  the  case  where  the  camera  undergoes  translation  only  along  the  optical  axis, 
the  expression  for  looming  for  points  in  the  X—Y  plane  becomes  very  simple,  i.e.. 


L 


e 

cotQ 


(42) 


An  alternative  method  for  computing  the  optical  flow  is  described  in  [53,64]. 

6.2.  DIRECT  CALCULATIONS  OF  LOOMING 
(FROM  RELATIVE  RATE  OF  EXPANSION) 

This  approach  deals  with  any  relative  motion  between  the  camera  and  the  object 
It  is  based  on  measuring  the  looming  at  the  image  region  (or  object)  level  (rather  than 
the  pixel  level).  The  relative  rate  of  expansion  of  an  object  in  the  image  is  propor- 
tional to  the  looming  as  defined  in  section  2. 

6.2.1  THE  3-D  BALL  SCENARIO 


The  concept  of  measuring  looming  from  the  relative  rate  of  expansion  can  be 
explained  using  a 3-D  ball  example.  Figure  12  shows  an  observer  moving  relative  to  a 
3-D  ball.  As  a result,  the  projection  of  the  ball  on  the  retina  may  be  different  in  size  at 
two  different  time  instants  t±  and  t2-  Let  the  relative  speed  along  the  ball-observer 
direction  be  It  I,  the  radius  of  the  ball  r,  and  the  distance  from  the  center  of  the  ball  to 
the  observer  R . At  t-t\ 


sinp  = ~ (43) 

K 

After  A t=t2-t\,  i*e.,  at  t=t2: 

Sin(|}+Ap)  = (44) 

For  infinitesimally  small  At  and  by  using  lim  — — = 1+x  the  last  equation  becomes 

x->o  l -x 

sin(p+Ap)  = X (45) 

By  subtracting  Equation  (43)  from  Equation  (45),  then  dividing  by  At,  and  letting 


- 14- 


At— >0  we  obtain: 


dsinp 

dt 


r Itl 
R 2 


(46) 


dR 


Using  the  following  relations:  --  = sinp,  •*==  = - and  = cosp  ^-  in 

Equation  (46)  we  obtain: 


dR  d§ 

dt  _ _ dt 
R tanP 

and  the  expression  for  looming  becomes: 

L = " 4=  "At  = “In(smp) 

R tanP  dt  K 


(47) 


(48) 


where  dot  denotes  derivative  with  respect  to  time. 

This  means  that  the  looming  can  be  measured  using  p and  P relative  to  the  center 

A 


of  the  ball. 


tanP 


contains  the  information  on  the  relative  expansion  of  the  projected 


ball  during  fixation  at  the  center  of  the  ball.  This  also  means  (as  detailed  later)  that  a 
retina  which  is  constructed  in  a logarithmic  fashion,  i.e.,  the  pixel  length  Ap  is  propor- 
tional to  tanP,  will  measure  looming  in  a linear  fashion  independently  of  the  size  of 
the  object  (as  long  as  the  retina  fixates  at  the  center  of  the  object). 

The  above  derivation  of  looming  is  based  on  a 2-D  angle  measurement  of  a pro- 
jection of  a ball.  However,  the  extension  to  the  projected  area  of  a ball  is  simple.  Rela- 
tive change  of  area  is  also  proportional  to  looming.  We  shall  show  next  that 


dA 

dt 

2A 


dR 

dt 

R 


- L 


(49) 


where  A is  the  projected  area  of  the  ball  on  an  spherical  image. 

2 ft  2 

The  area  occupied  by  a projected  ball  on  a spherical  image  is  A = 4rcsin  (-F-)ri 
where  P is  as  defined  in  Figure  12,  and  is  the  radius  of  the  sphere  of  the  retina. 
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is  constant  By  computing 


(divided  by  2),  we  obtain: 


dA 
dt_ 
2A 
dA 
cU_ 
2 A 


which  is  the  relative  change  of  the  projected  area 


dp 

— ~r-.  For  small  (3, 

2tan|- 

2 


it 


dt 


it 

_ dt 
tanp 


which  has 


been  shown  (Equation  (48))  to  be  the  expression  for  looming.  Thus  the  relative  change 
of  the  projected  area  can  be  used  to  approximate  looming  values  using  Equation  (49). 

6.2 2 THE  2-D  PLATE  SCENARIO 


We  describe  how  to  measure  looming  of  a flying  2-D  circular  plate  undergoing  an 
arbitrary  six-degree-of-freedom  motion.  Figure  13  shows  a sequence  of  images  of  a 
two-dimensional  plate.  Note  that  the  change  in  the  projected  area  cannot  be  used  to 
measure  looming.  However,  at  each  instant  of  time  there  is  a plane  which  is  perpendic- 
ular to  the  line  that  connects  the  pinhole  point  of  the  camera  with  the  center  of  the 
plate  (Figure  14).  The  intersection  between  the  plane  and  the  plate  is  a line  (except  for 
one  singular  case  where  the  intersection  is  a circle)  whose  projection  is  visible  to  the 
camera.  Since  this  line  (which  is  also  a diameter  of  the  plate)  is  perpendicular  to  the 
line  that  connects  the  camera  with  the  plate,  it  will  almost  have  the  longest  projection 
(compared  to  all  other  diameters’  projections).  The  projection  of  this  line  can  be  meas- 
ured. The  relative  change  in  length  of  this  line  can  be  used  as  a measure  for  looming. 
Similar  computations  as  described  previously  for  a section  of  a ball  hold  here. 

6.23  OTHER  OBJECTS 

The  problem  with  this  approach  is  that  in  many  cases  it  is  difficult  or  even  impos- 
sible to  measure  looming  using  the  described  method  since  a real  environment  is  not 
filled  with  3-D  balls  or  circular  plates.  In  several  cases,  however,  it  is  possible  to  get 
good  approximations  of  looming  especially  when  points  lie  near  the  plane  which  is  per- 
pendicular to  the  optical  axis. 

6.3  QUALITATIVE  MEASUREMENT  OF  LOOMING 

Looming  can  be  measured  in  qualitative  terms.  This  means  that  instead  of 
measuring  the  exact  value  of  looming,  we  may  be  interested  if  the  looming  value  is 


- 16- 


above  or  below  a threshold  level.  Alternatively,  we  may  be  interested  to  check  if  the 
looming  value  is  within  a specific  margin.  Figure  15  shows  what  we  mean  by  qualita- 
tive looming  (see  also  [33]).  In  Figure  15(a),  3-D  space  is  divided  into  two  regions. 
One  consists  of  all  points  inside  an  equal  looming  sphere,  and  the  second  region 
corresponds  to  all  points  on  or  outside  the  equal  looming  sphere.  Points  inside  this 
sphere  are  considered  to  be  in  a "Danger  Zone"  since  they  produce  "above  threshold" 
values  of  looming,  while  points  on  or  outside  the  ball  are  in  a "Safety  Zone"  since  they 
produce  "below  threshold"  values  of  looming. 

A more  quantitative  division  of  3-D  space  is  shown  in  Figure  15(b).  Here  the 
space  is  divided  into  several  regions,  according  to  the  range  of  looming  values  of 
points  in  each  region.  We  called  these  regions  "Danger  Zone",  "High  Risk  Zone",  and 
"Safety  Zone". 

6.4  ON  THE  MEASUREMENT  OF  LOOMING  WITH  A LOGARITHMIC  SEN- 
SOR 

In  this  section  we  show  that  it  is  "natural"  to  use  a logarithmic  sensor  to  measure 
looming.  (The  results  are  highly  related  to  neurobiologists’  observations  about  retinal 
mapping  [61]). 

What  is  a "logarithmic  sensor"?  If  the  camera  has  a multiresolution  imaging  chip, 
where  the  pixel’s  radial  size  is  approximately  proportional  to  its  distance  from  the  opti- 
cal axis,  then  it  is  called  "logarithmic".  Figure  16  shows  a log-polar  retina.  This  is 
different  from  a linear-polar  retina  (Figure  17)  where  the  pixel  radial  length  is  constant 
(We  wrote  "approximately  proportional"  since  it  is  not  possible  to  measure  uniquely 
the  distance  from  a finite  length  pixel  to  the  optical  axis.)  Similarly,  for  a spherical 
image,  if  the  pixel’s  angular  size  is  proportional  to  the  angle  (or  a function  of  the  angle 
e.g.,  sin(.))  between  the  optical  axis  and  the  line  that  connects  the  center  of  the  sphere 
with  the  pixel,  it  is  also  called  a multiresolution  logarithmic  retina. 

The  relevant  mathematical  expression  for  the  spherical  retina  (for  the  continuous 
case)  where  the  resolution  is  proportional  to  the  angle  from  the  optical  axis  is: 

P 


dlnP  = 


(50) 
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where  P is  the  angle  between  the  optical  axis  and  the  line  that  connects  the  center  of 
the  sphere  with  a particular  point  on  the  image  sphere.  In  practice  since  pixel  size  AfJ 
is  finite  this  expression  becomes: 

AlnP  = ^ (51) 

P 

Another  example  for  a logarithmic  retina  is  where  dp  is  proportional  to  tanp.  In 
this  case  the  relevant  mathematical  expression  is: 

(52) 

and  when  the  pixel  size  Ap  is  finite.  Equation  (52)  becomes: 

Aln(sinp)  = Mi  (53) 


We  have  shown  in  section  6.2.1  that  for  the  3-D  ball  scenario  (given  that  the  opti- 
cal axis  of  the  camera  passes  through  the  center  of  the  ball)  the  looming  can  be 

obtained  from  the  ratio  — 


tanp 


(Equation  (48))  independently  of  the  camera  rotation, 


where  P is  the  angle  between  the  center  of  the  ball  and  a point  on  the  circular  edge  of 
the  ball.  This  means  that  by  using  the  ln(sinp)  retina  (Figure  16,  as  opposed  to  the 
linear  retina  of  Figure  17)  it  is  possible  to  measure  looming  from  the  expansion  of  the 
projection  of  a ball  in  a simple  way  even  if  rotation  is  involved  ( as  long  as  the  optical 
axis  passes  through  the  center  of  the  ball,  i.e.,  the  camera  "fixates"  at  the  center  of  the 
ball).  In  other  words,  by  constructing  a retina  as  mathematically  described  in  Equation 
(52)  for  the  continuous  case,  or  as  in  Equation  (53)  for  the  discrete  case,  then  motion 
of  a point  in  the  image  from  pixel  ring  i to  pixel  ring  i+ 1 corresponds  to  the  same 
looming  value  as  motion  of  a point  in  the  image  from  pixel  j to  pixel  j+ 1,  and  the 
same  as  n times  less  the  looming  of  a change  from  pixel  k to  k+n , for  any  positive 
integers  ijjc,n.  In  other  words,  the  In  (sin ) retina  measures  looming  during  fixation  in 
a linear  way! 

Note  that  the  expression  for  looming  of  a 3-D  ball  (48)  is  equivalent  to  the 
expression  obtained  for  optical  flow  (42)  for  points  on  3-D  equal  looming  spheres  when 
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the  camera  undergoes  translation  only  along  the  optical  axis.  (0  in  Equation  (42)  is 
measured  from  the  X axis  which  is  perpendicular  to  the  optical  axis,  and  p in  Equation 
(48)  may  be  measured  relative  to  the  optical  axis.  Expressions  (42)  and  (48)  become 
identical  when  substituting  p = 9O°-0  and  p=-0.)  Also,  the  relative  change  in  the 
object’s  size  and  the  corresponding  change  in  the  pixels’  number  on  the  In  (sin)  retina 
are  proportional  as  can  be  seen  from  Equations  (48)  and  (49)).  So  far  the  existing 
literature  has  shown  how  a logarithmic  retina  can  be  used  for  simplifying  (i.e.,  lineariz- 
ing) the  measurement  of  the  optical  flow  and  time-to-contact  from  optical  flow  for  a 
camera  undergoing  translation  only  [65,68,69] . Here  we  showed  that  the  same  retina 
can  be  used  to  simplify  the  measurement  of  looming  of  a point  in  translational  motion 
and  for  fixation  on  the  center  of  a ball.  Recently  ([78]),  several  other  advantages  of  the 
logarithmic  retina  have  been  shown. 

The  derivation  here  is  an  extension  of  existing  results  and  may  be  another  advan- 
tage for  using  fixation.  Using  this  particular  logarithmic  mapping,  the  measurement  of 
looming  becomes  independent  of  the  object’s  size,  and  looming  becomes  linearly 
mapped,  i.e.,  different  size  balls  located  on  the  same  equal  looming  surface  result  in  the 
same  relative  change  in  pixels  of  the  logarithmic  retina.  Figure  18  shows  two  different 
size  balls  which  lie  on  the  same  equal  looming  sphere.  Even  though  their  projections 
on  the  retina  are  different,  their  relative  rate  of  expansion  is  the  same.  Using  a fixating 
logarithmic  retina  (Figures  19  and  20)  (as  described  previously)  their  expansion  in 
terms  of  number  of  pixels  is  the  same. 

7.  LOOMING  AS  A FUNCTION  OF  TIME 

So  far  we  have  dealt  with  instantaneous  looming.  We  now  examine  looming  as  a 
function  of  time.  As  shown  in  Section  4.2  equal  looming  surfaces  are  spheres.  The 
physical  size  of  each  sphere  is  proportional  to  the  instantaneous  translational  velocity, 
the  centers  of  the  spheres  lie  on  a line  along  the  instantaneous  velocity  vector,  and  the 
observer  lies  on  the  sphere  surface.  Also,  the  spheres  are  independent  of  the  instan- 
taneous rotational  parameters. 

Figure  21  shows  what  happens  to  these  spheres  as  a function  of  time.  Note  that 
their  centers  always  lie  on  a line  along  the  instantaneous  translational  vector  and  not  on 
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the  optical  axis.  Figure  22  examines  these  spheres  for  a particular  point  "A"  in  space. 
Assume  a camera  moving  in  constant  speed  along  a straight  line.  As  the  camera 
approaches  the  point  "A"  (for  f<f4),  the  equal  looming  spheres  containing  this  point 
shrink  i.e.,  produce  higher  values  of  looming.  At  a 0 = 45°  angle  as  in  Figure  8 (i.e.,  at 
time  instant  t=t4)  the  sphere  containing  point  "A"  gets  its  minimum  size  which  means 
that  the  looming  value  is  maximum.  Then,  when  the  camera  translational  vector  is  per- 
pendicular to  the  line  that  connects  the  point  "A"  with  the  camera  pinhole  point  (time 
instant  t=t5),  the  sphere  becomes  a plane,  i.e.,  the  point  "A"  produces  zero  value  of 
looming.  At  0 = -45°  (at  f=f6)  the  looming  value  is  minimum,  and  as  the  camera  con- 
tinues to  move  (t  >t6)the  looming  value  approaches  zero. 

Figure  23  shows  the  values  of  looming  of  several  points  as  a function  of  time. 
Figure  23a  shows  a camera  that  undergoes  both  translation  and  rotation.  We  examine 
the  looming  values  for  four  different  points:  1,2,3,  and  4.  The  looming  values  are  plot- 
ted in  Figure  23b.  At  time  instant  t = tl  the  looming  values  for  all  four  points  are 
positive,  and  the  looming  value  of  point  2 gets  its  maximum  value.  At  t = t2,  the 
looming  value  of  point  3 gets  its  maximum  value  since  the  angle  formed  by  the  instan- 
taneous translation  vector  and  the  line  that  connects  the  pinhole  point  of  the  camera  and 
point  3 is  45°.  At  time  close  to  t = f3  the  looming  produced  by  point  4 gets  a very 
high  positive  value  (which  changes  to  a very  high  negative  value  at  the  time  instant 
immediately  after  t = r3).  At  t = f3  points  1,  2,  and  3 are  all  on  the  same  equal 
looming  sphere  and  point  1 produces  a maximum  looming  value.  Note  that  the  loom- 
ing values  of  points  1,  2,  and  3 are  the  same  at  t = f3  as  shown  in  Figure  23b  since 
they  lie  on  an  equal  looming  sphere. 

Shortly  after  t = t3  point  3 produces  zero  looming  value.  At  t = r4  point  3 pro- 
duces minimum  value  of  looming  and  point  2 produces  zero  looming  value.  At  t = t$ 
point  2 produces  minimum  value  of  looming  and  point  1 produces  zero  value  of  loom- 
ing. At  t -t 6 points  1,  2,  3,  and  4 produce  the  same  value  of  looming  and  point  1 
produces  minimum  value  of  looming.  Note  that  looming  values  are  independent  of  the 
rotation  of  the  camera.  (The  camera  in  Figure  23a  was  chosen  to  fixate  on  point  2.) 
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The  case  where  a point  moves  relative  to  a camera  with  constant  value  of  looming 
can  be  described  in  two  basic  ways.  In  Figure  24a  a point  is  moving  in  the  XY  plane. 
At  each  instant  of  time,  e.g.,  tx,  t2,  1 3,  and  r4,  the  point  is  on  a physically  different 
sphere,  but  since  the  looming  values  produced  by  the  point  are  the  same  for  all  time 
instants  the  point  appears  on  the  same  equal  looming  sphere.  In  Figure  24b  the  coordi- 
nates are  scaled  by  a factor  of  Note  that  the  relative  translational  velocity  of  the 
point  changes  in  order  to  keep  a constant  looming  value  at  all  time  instants. 

8.  MORE  ON  LOOMING 

8.1  LOOMING  AND  THE  TIME-TO-CONTACT 

As  mentioned  earlier,  looming  is  different  from  the  time-to-contact.  It  is  related 
more  to  the  "two  dimensional"  time-to-contact  concept  [21].  According  to  Lee  [9]  any 
point  that  lies  on  a plane  which  is  perpendicular  to  the  instantaneos  translational  motion 
direction  of  the  camera,  will  produce  the  same  value  of  "time-to-contact"  Tc  (assuming 
that  the  optical  axis  coincides  with  the  direction  of  motion).  This  means  that  the  time- 
to-contact  deals  with  depth.  The  derivation  in  [9]  is  valid  only  for  rectilinear  motion 
(with  no  rotation)  of  the  camera.  Only  recently  [25,26]  has  it  been  extended  to  a more 
general  motion  of  the  camera. 

One  problem  with  the  time-to-contact  approach  is  that  points  which  lie  on  a single 
perpendicular  plane  but  are  located  far  away  from,  or  close  to,  the  camera  produce  the 
same  value  of  Tc  even  though  they  are  not  equally  relevant  to  malting  vision-based 
behavioral  decisions.  The  looming  value  of  a point  is  related  to  range  rather  than  depth. 

Figures  25  and  26  illustrate  the  main  difference  between  the  time-to-contact  value 
of  a point,  and  its  looming  value.  All  points  that  lie  on  a vertical  plane  will  have  the 
same  time-to-contact  value.  However,  points  on  a sphere  in  front  of  the  camera  produce 
the  same  looming  value.  Points  1,2,  and  3 in  Figure  26  have  the  same  looming  value 
but  different  "time  to  contact"  values.  Points  2 and  4 have  the  same  time-to-contact 
value  but  different  looming  values.  Points  4 and  5 have  the  same  time-to-contact 
values  and  the  same  looming  values. 
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8.2  LOOMING,  ZOOMING,  AND  VIEWING 

Note  that  since  looming  deals  with  the  relative  change  of  range  the  looming 
values  obtained  from  zooming  or  non-zooming  cameras  are  the  same,  although  the 
images  are  different  in  these  two  cases.  Also  the  looming  perceived  by  a camera  (#1 
in  Figure  27)  is  the  same  as  that  perceived  by  other  cameras  (e.g.,  #2,  #3,  and  #4  in 
Figure  27)  that  observe  a monitor  on  which  images  from  camera  #1  are  displayed 
(regardless  of  the  distance  and  location  relative  to  the  monitor)  This  is  due  to  the  fact 
that  the  relative  change  in  the  area  is  the  same  from  all  points  of  view. 

9.  LOOMING  AND  THE  EQUAL  FLOW  CIRCLES 
9.1  THE  EQUAL  FLOW  CIRCLES 

This  section  deals  with  points  in  3-D  space  that  produce  the  same  value  of  optical 
flow.  For  a particular  motion  of  the  camera  point  with  a specific  value  of  optical  flow 
lie  on  a circle.  We  briefly  review  this  result. 

It  has  been  shown  in  previous  work  [63]  that  if  the  camera  motion  vectors  t and  CD 
are: 

t = ( u,v,of 

and 

(0=  (0,0,C)r. 


Then  the  optical  flow  expression  in  spherical  coordinates  (Figure  11)  is: 
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From  Equation  set  (54),  the  points  in  space  that  result  from  constant  0 (regardless  of 
the  value  of  <J>)  form  a cylinder  of  infinite  height  whose  equation  is 
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, as  displayed  in  Figure  28a, 


and  the  points  in  space  that  result  from  <j>=0  (regardless  of  the  value  of  0)  are  those  that 
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lie  on  (a)  a plane  whose  equation  is  Y-  — —X,  or  (b)  a plane  whose  equation  is  Z=0 

(i.e.,  the  XY  plane)  as  pictorially  described  in  Figure  28b.  The  intersection  of  the 
cylinder  with  the  planes  is  the  desired  solution  (Figure  28c),  i.e.,  the  points  in  space 
that  result  in  0 = constant  and  <j>  = 0 optical  flow  values.  Analytically,  the  following 
are  the  solutions  (disallowing  the  case  of  X - 0 and  Y - 0 which  corresponds  to  an 
anomalous  situation): 
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These  solutions  are  drawn  in  Figure  29.  Solution  (55)  is  an  equation  of  a circle  that  lies 


in  the  XY  plane.  The  radius  of  the  circle  is 
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is  at 


2(C  +0)  2(C  +0) 


. The  circle  is  tangent  to  the  camera  translation  vector  at 


the  origin. 

Solution  (56)  is  a straight  line  perpendicular  to  the  XY  plane  and  intersects  this 
V U 


plane  at  the  point 


C +0  C+0 

defined  in  solution  (55). 


r , 0 


. This  intersection  point  also  lies  on  the  circle 


The  meaning  of  these  solutions  is  the  following:  all  points  in  3-D  space  that  lie  on 
the  circle  or  the  line  described  by  solutions  (55)  and  (56)  and  which  are  visible  (i.e., 
unoccluded  and  in  the  field  of  view  of  the  camera)  produce  the  same  instantaneous 
optical  flow  0 and  zero  instantaneous  optical  flow  <j).  We  call  the  circle  on  which  equal 
flow  points  lie  the  Equal  Flow  Circle  (EFC).  Two  sets  of  EFCs  are  illustrated  in  Fig- 
ure 30.  Figure  30a  shows  EFCs  for  the  case  where  the  camera  undergoes  instantane- 
ous translation  only.  The  label  of  each  circle  represents  the  optical  flow  0 in  the  image 
that  corresponds  to  points  on  this  circle.  Here,  there  is  a straight  line  (a  circle  with  an 
infinite  radius)  that  corresponds  to  zero  flow  in  the  image.  Figure  30b  shows  EFCs  for 
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the  case  where  the  camera  undergoes  both  instantaneous  translation  and  rotation.  Here, 
there  is  a circle  with  finite  radius  that  produces  zero  flow  (9  = 0 in  the  image  domain). 
We  called  this  circle  the  Zero  Flow  Circle  (ZFC). 

9.2  THE  EFCs  and  ZFCs  AS  A FUNCTION  OF  TIME 

As  the  camera  moves  through  3-D  space,  the  EFCs  move  with  it  Figure  31  is  an 
example  of  a camera  path  with  some  EFCs.  At  each  instant  of  time,  the  radii  of  the 
EFCs  are  a function  of  the  instantaneous  motion  parameters  t and  go.  The  locations  of 
the  EFCs  are  such  that  they  always  contain  the  origin  of  the  camera  coordinate  system 
(the  same  as  the  camera  pinhole  point),  are  tangent  to  the  instantaneous  translation  vec- 
tor t,  and  are  perpendicular  to  the  instantaneous  rotation  vector  co.  Each  ZFC  lies  to  the 
left  or  right  of  the  translation  vector  depending  on  whether  the  instantanous  rotation  is 
positive  or  negative,  respectively. 

9.3  LOOMING  - EFCs  MAPPING 

As  described  above,  there  are  equal  looming  circles  in  the  X-Y  plane  which  are 
sections  of  the  equal  looming  spheres.  These  circles  are  orthogonal  to  the  EFCs.  This 
observation  suggests  a 2-D  mapping  that  consists  of  two  orthogonal  families  of  circles 
as  shown  in  Figure  32. 

A point  in  2-D  space  can  be  mapped  in  camera  coordinates  by  specifying  two 
numbers.  One  is  the  optical  flow  of  the  point,  and  the  other  is  the  looming  value  of  the 
point. 

Figure  33  and  34  show  how  a segment  in  the  new  domain  (segment  11,12,22,21  in 
Figure  33)  can  be  mapped  to  another  orthogonal  grid  (segment  11,12,22,21  in  Figure 
34).  In  Figure  34  the  horizontal  axis  corresponds  to  the  optical  flow,  and  the  vertical 
axis  corresponds  to  the  value  of  looming.  Note  that  the  vertical  line  for  which  9 = 0 
corresponds  to  the  Zero  Flow  Circle. 

Regions  for  obstacle  avoidance  and  other  behavior  related  tasks  can  be  defined 
using  the  looming-EFCs  domain.  In  Figure  35  a qualitative  partition  of  the  2-D  plane 
is  shown.  A region  called  "danger  zone"  can  be  used  to  qualitatively  detect  obstacles.  A 
more  quantitative  partition  is  shown  in  Figure  36  where  space  is  divided  into  "danger 
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zone",  "high  risk  zone",  and  "safety  zone". 

Clearly,  the  mapping  can  be  extended  to  3-D  using,  for  example,  equal  flow 
cylinders  and  equal  looming  spheres. 

10.  DISCUSSION 

In  this  paper  we  presented  a new  and  mainly  quantitative  approach  for  analyzing 
looming.  Using  the  concept  of  the  equal  looming  spheres  we  showed  how  space  can 
be  perceived  during  motion.  Several  methods  for  measuring  looming  of  simple  3-D 
objects  were  presented.  It  has  been  shown  that  using  a logarithmic  retina  the  looming 
computations  become  simpler.  We  also  showed  how  beneficial  fixation  can  be  to  sim- 
plify the  measurement  of  looming.  This  analysis  complements  the  qualitative  under- 
standing of  looming  and  fixation. 

The  equal  looming  spheres  can  be  thought  of  as  properties  of  space  rather  than 
properties  of  points  in  the  image  domain. 

The  coordinate  system  that  we  chose  is  a convenient  one.  However,  the  angular 
velocities  of  points  in  space  are  independent  of  their  representation  in  the  image.  Other 
image  coordinate  systems  may  be  chosen,  in  particular,  (for  practical  purposes)  the 
image  domain  may  be  planar.  (Obviously,  an  appropriate  conversion  from  the  spherical 
coordinate  system  should  be  used). 

The  approach  for  measuring  looming  using  relative  rate  of  expansion  is  quite  lim- 
ited. It  works  mainly  for  sphere  and  sphere-like  objects,  or  sections  of  these  objects. 

Computing  looming  from  image  flow  by  the  method  of  Horn  and  Schunck  [36] 

has  the  advantage  of  simplicity.  Unfortunately,  it  tends  to  produce  noisy  and  inaccurate 

results.  There  are  four  sources  of  noise  and  errors:  First,  the  sampled  data  system 

dl  dl  i dl  , . , XT 

approximations  to  — , — and  , are  subject  to  digitization  noise  and  Nyquist 

dt  dQ  d <{> 

dl 

sampling  frequency  limitations.  Second,  except  in  the  vicinity  of  brightness  edges,  — ■ 

dt 

is  small,  and  may  be  zero.  Division  by  small  numbers  magnifies  errors.  Third,  the  sen- 
sitivity of  photodetectors  in  any  array  is  not  uniform.  The  difference  in  signal  from 
two  adjacent  photo-detectors  is  thus  not  necessarily  an  accurate  measure  of  the 
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difference  in  illumination.  Thus  range  from  image  flow  by  the  Horn  and  Schunck 
method  tends  to  be  inaccurate  for  smooth  images,  and  unreliable  for  sharply  focused 
natural  images. 

Computing  looming  from  image  flow  by  the  method  of  cross-correlation  [53]  can 
be  much  more  accurate  and  noise  free.  It  is,  however,  more  complicated  and  computa- 
tionally intensive. 

11.  FUTURE  WORK 

Future  work  will  focus  on  measures  of  performance  of  the  above  algorithms  in  an 
environment  filled  with  a variety  of  natural  objects.  Tests  will  be  made  that  simulate 
tasks  such  as  obstacle  avoidance.  We  plan  to  exploit  the  equal  looming  surfaces  con- 
cept in  a vision  based  navigation  algorithm  for  a real-time  robot  system. 
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Figure  2(a):  Projection  of  a Ball  at  Two  Time  Instants 
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Figure  2(b):  Projection  of  a Ball  at  Two  Time  Instants 


-36- 


Figure  3:  Motion  Relative  to  a Point 
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Figure  4:  Motion  Parameters 
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Figure  6:  Equal  Looming  Sphere 
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Figure  7: 


Equal  Looming  Spheres 
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Figure  8: 


Equal  Looming  Spheres 
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Figure  9:  Looming  for  Different  Angles 
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Figure  10:  Camera  Coordinate  System 
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Figure  11:  Image  Domain 
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Figure  12:  Looming  Computations 
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Figure  13:  Projections  of  2-D  Plate 
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Figure  14:  Moving  2-D  Plate 
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Figure  15(a):  Qualitative  Looming 
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Figure  15(b):  Qualitative  Looming 
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Figure  16:  Logarithmic  Retina 
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Figure  17:  Linear  Retina 
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Figure  18:  Looming  and  Fixation 


Figure  19:  Looming  and  Fixation 
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Figure  20:  Looming  and  Fixation 
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Figure  21:  Equal  Looming  Spheres  as  a Function  of  Time 
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Figure  23: 


Looming  as  a Function  of  Time 
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Figure  24(a):  Keeping  Constant  Looming  Value 
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Looming  Value 
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Figure  25:  Looming  and  Time-to-Contact 
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Figure  27:  Observing  a Screen 
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Figure  28:  (a)  Equal  0 Cylinders 

(b)  Equal  jf=  0 Planes 

(c)  IntersectionBbetween  0 Cylinders  and  jf=0  Planes 
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Figure  29: 


Pictorial  Description  of  Solutions 
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Figure  30:  Optical  Flow  Values  Due  to  Camera  Motion 

(a)  For  Translation  Only 

(b)  For  Translation  and  Rotation 
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Figure  31: 


EEC'S  as  a Function  of  Time 
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Figure  32 


EFC'S-Looming  Mapping 
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Figure  33: 


EFC'S-Looming  Mapping 
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Figure  34:  Isometric  Mapping  of  Looming  and  EFC'S 
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Defining  Obstacles  Using  Looming 
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Defining  Obstacles  Using  Looming 
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